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Definitional  discussions,  especially  in  short  papers, 
may  appear  to  be  carried  on  at  the  expense  of  "doing  what 
the  paper  is  about."  However,  I prefer  to  view  the  clarifi- 
cation of  definitional  problems  as  initial  investments  which 
yield  high  (and  immediate)  rates  of  return  in  the  form  of 
both  increased  clarity  and  conciseness.  Therefore  the  first 
section  of  this  paper  will  involve  stating  as  precisely  as 
possible  the  questions  to  be  discussed  in  the  following 
sections.  In  addition,  though  no  less  important,  the  next 
section  should  serve  to  provide  a general  context  in  which 
the  specific  topics  I will  be  treating  become  interesting. 

I Problems  and  Context 

The  task  of  this  paper,  as  is  evident  from  its  title  is 

to  appraise  the  methodological  practices  used  in  the  Inter- 

Nation  Simulation  Project  (INS).  Such  a task  almost  requires 

quoting  the  remarks  with  which  R.F.  Harrod  (1938)  began  a 

"methodological  essay." 

Exposed  as  a bore  the  methodologist  cannot 
take  refuge  behind  a cloak  of  modesty.  On 
the  contrary,  he  stands  forward  ready  by  his 
own  claim  to  give  advice  to  all  and  sundry, 
to  criticise  the  work  of  others,  which,  whether 
valuable  or  not,  at  least  attempts  to  be  con- 
structive; he  sets  himself  up  as  the  final 
interpreter  of  the  past  and  dictator  of  future 
efforts . 

Such  observations  are  especially  important  as  one  attempts 
to  "assess"  such  a major  and  obviously  constructive  project  as 
INS.  The  enormity  of  the  technical  output  generated  in  INS 
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related  research  can  be  begun  to  be  sensed  as  one  reads  the 
annotated  bibliography  of  Leserman  (1972).  That  a political 
scientist  might  undertake  to  critique  INS  from  a methodological 
perspective  is  itself  an  indication  of  the  successful  accom- 
plishment of  Guetzkow's  goal  of  developing  a "college  of  simu- 
lators" within  political  science.  Indeed,  one  measure  of  the 
success  of  an  innovative  project  such  as  INS  might  well  be  the 
variety  and  quality  of  criticism  it  draws. 

Since  this  paper  involves,  in  part,  "doing  methodology," 
it  will  be  useful  to  clarify  how  I will  be  using  the  term. 

This  is  especially  important  since  many  behavioral  scientists 
appear  to  use  the  two  terms  "methodology"  and  "technique"  as 
synonyms.  I would  like  to  distinguish  the  two  and  suggest  the 
following.  Methodology  deals  with  questions  of  justification. 

i 

"The  method  of  a science  is,  indeed,  the  rationale  on  which  it 

f 

bases  its  acceptance  or  rejection  of  hypothesis  or  theories 
(Rudner,  1966:5)."  Techniques,  on  the  other  hand,  are  ways  of 
generating  information  which  is  used  in  evaluating  a hypothesis 
or  theory.  Thus  methodological  positions  will  influence  choice 
of  technique.  For  example,  a Hempelian  scientist  would  probably 
argue  that,  for  methodological  reasons,  the  technique  of  crystal 
ball  gazing  is  an  inappropriate  technique  for  the  scientist. 

Techniques  then  are  ways  of  generating  evidence  and  methodology 
provides  the  rules  according  to  which  the  evidence  is  admitted 
and,  if  admitted,  evaluated. 
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The  distinction  I am  drawing  between  technique  and 
methodology  is  important  for  several  reasons.  First,  it 
suggests  that  we  can  not  critique  the  use  of  a particular 
technique  without  first  specifying  the  methodological 
perspective  from  which  our  critique  is  directed.  Second,  and 
perhaps  a corollary  of  the  first,  techniques  which  are  inappro- 
priate from  one  methodological  perspective  can  become  appropri- 
ate under  another.  Thus  in  appraising  the  techniques  used  in 
INS  I must  be  concerned  with  questions  of  methodology  choices. 
Indeed,  as  will  be  argued,  different  methodological  choices 
may  be  made  depending  upon  the  purpose(s)  of  the  investigation. 
Third,  and  perhaps  least  important  for  this  paper,  distinguishing 
technique  from  methodology  allows  the  possibility  that  not  being 
able  to  use  all  the  techniques  of,  say  physics,  does  not  preclude 
the  use  of  a methodology  appropriate  to  physics.  To  summar- 
ize, all  three  reasons  support  the  position  that  question  of 
technique  cannot  be  totally  separated  from  the  philosophical 
questions  surrounding  choice  of  method. 

Since  whatever  else  INS  may  be  viewed  as  having  done, 

it  certainly  involves  the  use  of  simulation-  in  the  investigation 

* ^ 

of  international  relations  phenomena,  it  is  important  to  define 
what  is  meant  by  the  term  simulation.  In  doing  this,  it  will  be 
helpful  to  discuss  the  related  concepts  of  model  and  theory. 

After  providing  explications  and  definitions  for  these  terms, 

I will  then  attempt  to  pose  the  specific  questions  the  remainder 
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of  this  paper  will  address. 

Harold  Guetzkow  has  defined  a simulation  as 

"An  operating  representation  in  reduced 
and/or  simplified  form  of  relations  among 
social  units  by  means  of  symbolic  and/or 
replicate  component  parts  (Guetzkow,  1959, 
p.  184)." 

The  use  of  the  adjective  "social"  restricts  the  class 
of  simulations,  but  to  generalize  the  definition  one  need  only 
remove  social.  Indeed,  this  definition  suggests  that  there 
are  three  elements  which  need  be  specified  in  talking  about 
a simulation.  First  there  is  the  system  to  be  simulated,  S. 
Second  there  is  the  representation  of  that  system,  M.  Third 
there  are  the  statements  according  to  which  the  representation 
is  described  and/or  "operates,"  T.  These  statements  might  be 
in  Fortran,  PLl,  English,  Chinese,  etc.  We  can  think  S,M,  and 
T as  being  sets.  That  is,  S is  the  set  of  all  referrents  (or 
systems  to  be  simulated);  M is  the  set  of  all  representations 
of  S;  and  T is  the  set  of  all  statements  according  to  which  the 
representations  "operate."*  Notationally  we  can  write: 


S: 

{ 

Si, 

®2’ ' ' 

. .Sn) 

M: 

{ 

m J , 

in  2 1 • * 

T: 

{ 

^2'  * ■ 

••tp> 

Any  specific  simulation  can  then  be  identified  with  an  ordered 
triple  which  tells  us  which  member  of  S is  being  represented  by 
what  member  of  M in  terms  of  what  statements  in  T.  These  dis- 
tinctions are  use  are  useful  because  they  point  out  that  under  j 

* This  discussion  is  based  in  part  upon  Zeigler , (1970)  j 
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the  Guetzkow  definition  changing  any  one  of  the  three  elements 
gives  us  a new  simulation.  Thus  in  predicating  statements  of 
INS  we  must  be  as  careful  as  we  can  in  specifying  each  of  the 
three  components.  All  too  often  the  literature  seems  to  equate 
INS  only  with  an  element  of  M.  As  will  be  argued  below,  this 
has  the  implication  of  ignoring  the  impact  changes  in  the  sCS 
might  have  upon  the  resolution  of  methodological  and  technical 
questions. 

However  before  I can  make  these  arguments  I must  first 
explicate  more  completely  what  elements  of  S and  M look  like. 
Elements  of  S are  difficult  to  visualize  in  any  concrete  way 
since,  by  visualizing  them  they  become  represented.  The  elements 
of  S are  the  "realities"  discussed  in  Guetzkow  (1968)  or,  equi- 
valently the  "reference  systems"  discussed  in  Hermann  (1967, 
p.  220).  There  is  no  reason  to  require  that  these  realities  be 
in  any  sense  completely  known.  Indeed  it  is  precisely  because 
they  are  complex  (and  perhaps  partially  unknown)  that  we  choose 
to  represent  them  as  a part  of  a simulation  effort  (an  elegant 
argument  for  this  point  is  found  in  Naylor,  1971,  pp.  2-10  ). 

An  adequate  representation  consists  of  specifying  the 
objects  making  up  the  representation  and  the  relations  which 
are  defined  on  these  objects.  That  is,  a representation  con- 
sists of  a well  specified  world.  'This  notion  of  representation 
is,  I believe,  the  one  intended  by  Guetzkow  when  he  writes, 

5 "Simulations  in  international  relations  attempt  to  represent 

f 

i the  on-going  international  system  or  components  there  of,  such  as 
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world  alliances,  international  organizations,  regional  trade 
processes,  etc.  (1969,  285),"  and  by  Hermann,  "A  simulation 
or  game  is  a partial  representation  of  some  independent  system 
(1967,  216)". 

A representation  is  a mathematical  structure  (a  collection 
of  objects  and  relations)  and  one  can  ask  questions  of  possible 
relations  between  the  representation  and  the  referrent  reality. 
For  example,  is  there  a homomorphism  between  s^  and  m^ ; an  iso- 
morphism; or,  more  generally,  what  sorts  of  morphisms  ("general- 
ized" mappings)  might  one  be  willing  to  assert  to  obtain  between 
Sj  and  m^ . Loosely  speaking  these  might  be  termed  questions 
of  external  validity.  Answers  to  these  questions  will  require 
an  examination  of  possible  purposes  of  INS. and  will  be  considered 
later  in  this  essay. 

Simply  representing  a referrent  reality  (i.e.,  picking 
an  mfM)  is  not  enough  to  provide  us  with  a simulation.  The 
representation  will  often  be  too  complex  to  study  directly. 
Therefore  we  will  be  interested  in  seeing  what  sorts  of  beha- 
viors  are  produced  (generated)  by  the  representation.  This 
objective  requires  developing  a theory  of  the  representation. 

This  theory  will  take  the  form  of  statements  "about"  the  repre- 
sentation. In  the  case  of  a simulation,  the  theory  may  be  viewed 
as  a set  of  "operating  rules."  The  language  in  which  these 
operating  rules  or  statements  are  written  will  depend  in  part 
upon  the  mechanism  chosen  for  use  in  the  simulation.  In  an  all 
machine  simulation,  the  statements  might  all  be  in  Fortran,  or 
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Dynamo,  or  Simscript,  etc.  In  a simulation  combining  human 
subjects  and  a computer,  the  operating  rules  may  be  partly 
in  Fortran  and  partly  in  some  natural  language. 

Once  again,  in  any  specific  simulation  effort,  we  will  be 
led  to  ask  questions  as  to  possible  relations  to  assert  be- 
tween elements  of  T and  the  elements  of  S and  M.  Here,  however, 
our  concern  is  with  relations  between  structures  (m^  and  Sj) 
and  sets  of  sentences  Or,  more  generally,  one  concern 

is  with  the  relations  between  sentences  and  objects  and  rela- 
tions "referred  to"  by  those  sentences.  Such  relations  may 
be  termed  semantic  (Tarski,  1944).  An  example  of  such  a seman- 
tic relation  is  truth.  Are  the  sentences  true  of  the  represen- 
tation? Are  they  true  of  the  referrent  reality?  Whether  or  not 
particular  sentences  (or  sets  of  sentences)  are  accepted  as  true 
is  a question  of  methodology.  What  it  means  to  assert  them  to 
be  true  is  a (non- trivial ) question  of  semantics. 

Thus  far  this  discussion  has  been  focused  upon  simula- 
tion. Perhaps  the  reader  has  sensed  that  what  is  involved  in 
doing  simulation  is  quite  similar  to  what  is  involved  in  doing 
theory  in  general.  Therefore  it  will  be  useful  to  quickly 
relate  what  has  been  done  so  far  to  the  commonly  encountered 
terms  theory  and  model.  Accomplishing  this  will  enable  me  to 
bring  in  more  general  methodological  points  as  this  essay  pro- 
gresses. 

The  particular  set  of  sentences  or  statements  (i.e.,  t6T) 
making  up  the  simulation  are  roughly  equivalent  to  a technical 
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sense  of  theory.  They  are  sets  of  sentences  which  are 
asserted  to  be  true  of  some  world.  Further,  these  sets  of 
sentences  will  generally  (always  if  an  artificial  programming 
language  is  used)  have  some  preassigned  logical  framework  or 
"calculus  axioms"  (e.g.,  first  order  predicate  calculus,  prob- 
ability theory,  etc.).  These  axioms  allow  us  to  investigate 
the  implications  of  subsets  of  our  set  of  theory  statements. 

A model  for  a theory  is  that  "thing"  which  makes  the 
sentences  in  a theory  true.  In  theorizing  we  generally  want 
to  order  or  account  for  some  aspects  of  a perceived  reality 
or  referrent  system.  Thus  we  represent  reality  in  terms  of 
some  posited  objects  and  relations.  Whether  or  not  these  posited 
objects  and  relations  indeed  represent  reality  is  of  course  in 
many  senses  moot  and  is  certainly  contingent  upon  both  our  per- 
ceptual system  and  our  ability  to  make  and  hold  to  distinctions. 

However,  as  we  have  seen,  a collection  of  objects  and  rela- 
tions is  a mathematical  structure  and  not  a theory.  We  must 
write  down  some  sentences  describing  (i.e.,  which  are  true  of) 
this  structure.  These  sentences  I have  termed  a theory.  The 
underlying  structure  I will  call  a model  for  that  theory. 

More  specifically,  a mathematical  structure  m is  a set 
of  elements  (objects),  A =■  (a^^,  a2,...},  together  with  a 
set  of  relations  of  order  i, 
expressed 

m*  <A;  Pj  If  P2  2 , . . . n , . . . > . 
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A formal  language  L in  which  properties  of  M can  be 
expressed  will  consist  of  formulas  generated  by  a specified 
set  of  rules,  say  the  predicate  calculus,  from  an  alphabet 
consisting  of  relation  symbols  (R^,  R21  • • .).  variable  sym- 
bols (Xj,  X2,  . . .).  connectives  (“1,'^,-^,  . . .)  and  quan- 
tifiers (V  )•  Since  functions  and  constants  are  special 
kinds  of  relations,  function  symbols  (fj^,  f2,  . . .)  and  con- 
stant symbols  (Cj,  and  C2,  . . .)  will  also  be  used  in  L.  The 
language  L will  be  assumed  to  be  first  order,  that  is,  its  varia- 
bles range  over  the  elements  of  A (as  opposed  to  ranging  over 
the  subsets  of  A,  or  sets  of  subsets,  etc.)*  Sentences  in  L 
are  formulas  containing  no  free  variables. 

Let  T be  a set  of  axioms  in  a language  L.  If  is  a mapping 
of  constant  symbols  occurring  in  T into  the  set  of  objects  A, 
and  also  a mapping  of  relation  symbols  occurring  in  T into  the 
set  of  relations  in  M,  then  M provides  an  interpretation  of 
T under  (}> . If  this  interpretation  results  in  the  sentences  in 
T being  true,  then  M is  said  to  satisfy  T and  M is  a model  of 
the  axiom  set  T.  A model  for  a set  of  axioms  then,  is  a mathe- 
matical structure  which  is  used  to  interpret  the  axioms  in 
such  a way  that  the  axioms  are  true.* 

The  above  discussion  may  appear  needlessly  abstract. 

However,  its  generality  enables  us  to  do  several  things.  First 
it  makes  explicit  a relationship  between  the  simulation  enter- 


* 


This  discussion  is  taken  from  S.  Thorson  and  J.  Stever,  (1974). 
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^ prise  and  the  theory  enterprise.  To  be  specific,  under 

these  definitions,  simulations  become  a subset  of  theory. 

This  means  we  can  employ  (as  is  suggested  in  Guetzkow  (1968)) 
the  kinds  of  critical  methodological  tools  used  to  analyze 
scientific  theory  to  analyze  simulation  efforts  such  as  INS. 

Second,  I have  said  that  one  component  of  any  simulation 
is  an  explicit  representation  (mOl)  of  a referrent  system.  Thus 
simulations  belong  to  the  subclass  of  theories  with  explicit 
models.  By  a result  from  model  theory,  we  know  that  if  an 
axiomatic  deductive  theory  has  a model,  then  it  is  logically 
consistent  (i.e.,  it  is  non-contradictory).  This  point,  while 
subtle,  is  of  extreme  importance.  It  was  not  until  the  nine- 
j teenth  century,  for  example,  that  model  theory  was  employed 

to  show  that  negating  the  parallel  lines  axiom  of  Euclidean 
geometry  did  not  lead  to  an  inconsistency  (assuming  Euclidean 
geometry  itself  to  be  consistent). 

Third,  by  making  the  representation  (model)  explicit, 
it  is  possible  to  efficiently  investigate  results  of  slightly 
(or  grossly)  perturbing  the  representation.  This  kind  of  sen- 
sitivity testing  is  very  difficult  to  do  for  theories  without 
explicit  models.  That  INS  was  used  in  such  a way  is  documented 
by  the  number  of  INS  "variants"  which  have  been  produced  (e.g., 
Bremer  (1970),  Smoker  (1968),  Abt  and  Gordon  (1969),  and  Leavitt 
(1971)) . 
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Having  defined  simulation  and  related  this  definition 
to  that  of  theory  and  model  it  should  be  apparent  that  a 
methodological  critique  (under  my  use  of  methodology)  of  INS 
must  address  questions  of  the  utility  of  INS  in  the  develop- 
ment of  international  relations  theory.  To  be  fair  to  the 
INS  project,  such  an  appraisal  must  attempt  to  perform  this 
evaluation  within  the  context  of  the  projects  stated  theore- 
tical goals.  Therefore,  I will  organize  the  remainder  of  this 
essay  around  the  general  problem  of  validity  with  respect  to 
a variety  of  stated  theoretical  objectives. 
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II  Methodological  Critique 

One  of  the  great  strengths  of  the  INS  project  may  be  seen 
as  its  continuing  and  serious  concern  with  problems  of  valid- 
ation. This  concern  has  been  reflected  in  such  papers  as  Alger 
(1963),  Chadwick  (1967),  Crow  and  Noel  (1965)  Guetzkow 
(1966,  1967),  Hermann  (1967),  Hermann  and  Hermann  (1967),  Kress 
(1966) , McGowan  (1972),  Nardin  and  Cutler  (1967),  Noel  (1963), 
Raser,  Campbell, and  Chadwick  (1972),  Robinson  et  al  (1966), 
and  Verba  (1964).  Given  the  methodological  focus  of  this  essay, 
perhaps  two  of  the  most  important  of  these  papers  are  Guetzkow 
(1968)  and  Hermann  (1967).  The  Hermann  paper  is  something  of 
a classic  and  is  often  cited  both  within  and  outside  of  the 
l behavioral  sciences  as  a standard  piece  on  the  validation  of 

simulations . 

Hermann  (1967,  217)  argues  that  ''....validity  is  not  a 

singular  issue..."  and  that  "...we  may  more  accurately  refer 

to  multiple  validity  issues."  He  then  goes  on  to  identify 
i 

three  components  of  validity  issues. 

First  the  validity  of  an  operating  model  is 
affected  by  the  purpose  or  use  for  which  the 
game  or  simulation  nas  been  constructed.  What 
may  be  a relatively  valid  operating  model  for 
one  objective  may  be  strikingly  unsatisfactory 
for  another.  Second,  model  validation  can  be 
expected  to  vary  according  to  the  type  of  validity 
criteria  employed.  Third,  the  validation  issues 
will  be  significantly  altered  depending  on  whether 
human  participants  are  introduced  into  the  model 
217). 
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Guetzkow  (1969)  identified  three  general  purposes  for 
simulations  in  the  study  of  international  relations; 

Simulations  may  serve  in  three  ways  as 
formats  through  which  intellectuals  may 
consolidate  and  use  knowledge  about  inter- 
national relations:  (1)  Simulations  may 
be  used  as  techniques  for  increasing  the 
coherence  within  and  among  models,  enabling 
scholars  to  assess  gaps  and  closures  in 
our  theories;  (2)  Simulations  may  be  used 
as  constructions  in  terms  of  which  empiri- 
cal research  may  be  organized,  so  that  the 
validity  of  our  assertions  may  be  appraised; 

(3)  Simulations  may  be  used  by  members  of 
the  decision-making  community  in  the  devel- 
opment of  policy,  both  as  devices  for 
making  systematic  critiques,  through  "box- 
scoring" its  failures  and  successes,  and 
as  formats  for  the  exploration  of  alter- 
native plans  for  action  (1969,  286). 

These  three  purposes  might  be  labelled  "programmatic 
guide  *o  research,"  "description,"  and  "policy"  respectively. 

That  is,  in  the  first  the  concern  is  with  integrating  "islands 
of  theory"  and  organizing  "the  division  of  labor  more  coherently 
among  the  scholars  working  within  international  affairs  (1969: 
288)."  In  the  second,  the  emphasis  is  upon  the  degree  of  corre- 
spondence between  the  representation  and  the  reference  system, 
i.e.,  description.  In  the  third,  primary  focus  is  upon  policy 
planning  and  the  specification  of  alternative  futures.  Policy 
planning  may  be  viewed  as  involving  the  specification  of  prob- 
ability distributions  over  consequences  of  alternative  (feasible) 
policies.  For  the  purposes  of  this  essay  it  will  be  helpful 
to  separate  the  "policy"  objective  from  what  might  be  termed 
the  "design"  objective. 
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In  design  we  are  less  concerned  with  identifying  impact 
of  alternative  (presently)  feasible  policies  and  more  concern- 
ed with  identifying  new  structures  for  the  achievement  of 
particular  goals.  This  notion  of  design  is  compatible  with 
what  Guetzkow  (1966,  1969)  has  termed  "constructin  alternative 
futures".  As  he  points  out,  this  concern  with  design  is  one 
of  the  oldest  raditions  in  political  science.  And,  he  argues, 
simulation  is  an  important  tool  for  the  design  theorist.  One 
reason  for  this  importance  is,  as  we  have  already  seen,  due  to 
the  relative  case  with  which  the  impact  of  alternative  represent- 
ations (models)  can  be  investigated. 

A fifth  purpose  for  simulation  (and  INS  in  particular) 
which  is  often  cited  (e.g.,  Guetzkow  (1959))  is  that  of  educa- 
tion. INS  has  been  used  to  teach  principles  of  international 
politics  in  high  schools,  colleges  and  universities  and 
various  governmental  bureaucracies.  The  few  studies  I have 
seen  of  the  effectiveness  of  INS  versus  more  traditional  tech- 
niques (e.g.,  case  studies,  Robinson  (1961))  have  not  found 
INS  to  be  clearly  "superior".  However , given  the  lack  of 
agreement  on  appropriate  educational  objectives  (evei  within 
the  cognitive  domain)  the  question  of  the  relative  utility  of 
INS  type  simulations  for  students  with  certain  educational 
objectives  is  still  an  open  one. 

Leaving  aside  the  educational  purpose,  let  me  next  examine 
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* INS  along  each  of  the  first  four  objectives  in  somewhat  more 

detail.  These  four  were: 

1)  Programmatic  guide  to  research 

2)  Description 

3)  Policy 

4)  Design 

Programmatic  guide  to  research:  This  purpose  is  potentially 
one  of  the  most  important  for  large  scale  simulation  projects. 

Reasons  for  this  importance  have  been  brought  out  in  another 
context  by  Allen  Newell  (1972)  in  a very  interesting  analysis 
of  contemporary  experimental  psychology.  He  argues  "that  the 
two  constructs  that  drive  our  current  experimental  style  are  1) 
at  a low  level,  the  discovery  and  empirical  exploration  of  (dis- 
crete empirical  phenomena)  and  2)  at  the  middle  level,  the  for- 
mulation of  questions  to  be  put  to  nature  that  center  on  the 
resolution  of  binary  oppositions."  This  characterization  could 
be  equally  well  made  of  the  contemporary  empirical  study  of 
international  relations.  A look  at  recent  journal  articles  or 
convention  papers  illustrates  the  concern  with  discrete  phenomena. 

Further  these  investigations  are  often  driven  by  and  imbedded 
in  such  binary  distinctions  as  internal/external,  conflict/ 
cooperation,  rat ional/irrat ional , large/small,  open/closed, 
developed/underdeveloped,  etc.  Will  this  approach  to  science 
(Newell  terms  it  "playing  twenty  questions  with  nature")  work? 

It  may.  However,  "reality"  may  be  too  complex  to  yield  to  this 
approach.  That  is,  we  may  not  be  able  simply  to  add  up  answers 
to  these  "simple"  questions  to  got  a general  theory  of  inter- 
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national  relations.  An  alternative  to  playing  twenty  questions 
with  nature  is  to  construct  "complete  processing  models,"  i.e., 
large  scale  simulations  such  as  INS.  Such  simulations  allow  us 
to  examine  particular  phenomena  (e.g.,  nuclear  proliferation 
(Brody,  1963)  or  public  goods  and  alliances  Burgess  and  Robin- 
son (1969))  as  part  of  a general  problem  structure  (INS). 

INS  appears  to  have  been  moderately  successful  in  terms  of 
this  objective.  Numerous  studies  have  been  done  using  INS 
as  a representation  within  which  to  investigate  specific  hypo- 
theses. However,  before  we  can  begin  to  assess  the  utility  of 
these  studies  it  will  be  necessary  to  consider  the  related  ob- 
jective of  description. 

Description;  In  the  introductory  section  of  this  paper  I 
identified  three  elements  in  a simulation  - referrent  system  (S) , 
representation  or  model  (M) , and  statements  (T) . The  question  of 
descriptive  adequacy  is  a question  of  the  S-M  relationship. 

If  both  the  referrent  system  (reality)  and  the  model  are  viewed 
as  "black  boxes"  with  inputs  and  outputs,  the  problem  of  descrip- 
tive adequacy  can  be  posed  in  terms  of  the  morphism  (mappings) 
which  are  preserved  between  S and  M. 

Brodbeck  (1959)  in  her  essay  on  models  and  theories  suggests 
that  there  should  be  an  isomorphism  between  s and  m.  An  isomor- 
phism is  a term  used  to  denote  a mapping  between  two  structures 
such  that  there  is  a one  to  one  correspondence  between  the  ob- 
jects and  relations  of  the  first  and  the  second  structure.  Yet, 
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as  Guetzkow  (1968,  207)  points  out,  such  a requirement  is 
far  too  strict.  I would  even  go  beyond  this  and  argue  that 
such  a requirement  is  generally  undesirable.  A model  which 
was  isomorphic  to  reality  would  be  as  intractable  as  reality 
itself,  "...simulations...-  like  other  models  - are  always 
a simplification  of  their  reference  system  (Hermann,  1967: 
217)."  Thus  we  generally  ask  that  the  relation  between  the 
reference  system  (s)  and  the  model  (m)  be  a homomorphism 
(Guetzkow,  1968:  207).  Here,  at  an  intuitive  level,  rather 
than  requiring  a one-one  corresponce,  we  allow  many  elements 
and  relations  of  s to  be  mapped  into  those  of  m. 

However,  and  to  my  knowledge  this  has  not  been  explicitly 
considered  in  INS  related  research,  there  are  a variety  of 
morphisms  which  might  be  asserted  to  obtain  between  s and  m 
(e.g.,  see  Zeigler,  1970,  1971).  Three  such  morphisms  can  be 
termed  "behavior  preserving,"  "function  preserving,"  and 
"structure  preserving."  The  weakest  of  these  is  behavior 
preserving.  Here  the  concern  is  only  that  equivalent  inputs 
in  s and  m produce  equivalent  outputs.  Function  preserving 
morphisms  preserve  not  only  input-output  relations,  but  also 
internal  state  changes  (see  Arbib,  1969  for  a discussion  of 
state).  Finally  the  most  restrictive  morphism  - structure 
preserving  - preserves  (in  addition  to  input-output  relations 
and  state  transition  functions)  the  manner  in  which 
these  relations  and  functions  arise  out  of  local  coordinate 
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functions  (see  Zeigler,  1970;  pp  6ff).  While  I will  not 
discuss  structure  preserving  morphisms  in  this  paper,  the 
set  theoretic  view  of  simulation  developed  in  the  beginning 
of  this  essay  is  general  enough  to  pursue  such  investigations. 

The  first  two  of  these  morphisms  - behavior  preserving 
and  structure  preserving  - may  be  formalizations  of  what 
Guetzkow  meant  when  he  wrote: 

Some  homomorphy  may  exist  among  outputs  as 
well  as  between  the  very  processes  which  result 
in  such  outputs.  As  we  analyze  the  corres- 
pondences between  simulations  and  "realities" 
sometimes  an  internal  process,  like  the  repre- 
sentation of  the  decision-making  within  foreign 
offices,  helps  produce  an  outcome  of  some  vali- 
dity, such  as  the  constellation  of  internation 
alliances.  At  other  times  less  often  because  of 
lack  of  appropriate  research  an  internal  process 
will  be  judged  to  be  of  some  validity  because 
the  very  process  itself  has  some  congruence  with 
corresponding  processes  in  the  reference  data 
(1968,  207). 

A homomorphism  among  only  outputs  would  be  similar  to  the 
behavior  preserving  morphism,  while  a morphism  which  preserved 
outputs  as  well  as  "internal  processes"  would  be  a function 
preserving  morphism. 

Most  of  the  Hermann  (1967)  essay  seems  to  be  concerned 
with  identifying  criteria  for  being  able  to  assert  a behavior 
preserving  morphism  between  s and  m.  In  the  section  discussing 
criteria  for  assessing  the  "fidelity  with  which  a model  pro- 
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I 1)  Internal  validity 

2)  Face  validity 

3)  Variable  - parameter  validity 

4)  Event  validity 

5)  Hypothesis  validity 

It  will,  I think,  be  useful  to  consider  briefly  each  of  these 
for  each  has  implications  for  the  method  by  which  simulations 
are  constructed. 

"Any  exogenous  inputs  introduced  during  the  course  of 
the  game  are  held  constant  across  all  trials  or  runs.  The 
unexplained  variance  between  these  intended  replications  would 
provide  a measure  of  reliability  or  what  Campbell  (1957)  calls 
'internal  validity.'  When  the  structured  simulation  properties 
are  held  constant,  the  smaller  the  between-run  variance,  the 
greater  the  internal  validity  is  assumed  to  be  (Hermann,  1967: 
220)."  This  notion  of  internal  validity  or  reliability  seems 
to  me  to  be  somewhat  misleading.  First  of  all  note  that  the 
concern  here  is  not  with  a relation  between  s and  m.  Rather 
an  interest  is  purely  in  properties  of  m (or  perhaps  relations 
between  m and  t) . 

Hermann  seems  to  be  asserting  that  it  is  desirable  for  two 
runs  of  the  same  simulation  with  equivalent  input  to  show  equi- 
valent output  (response).  With  input  fixed,  he  wants  to  mini- 
mize between  run  variance.  However  if  the  simulation  is 
stochastic  (i.e.,  probabilistic)  either  due  to  the  explicit 
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♦ use  of  pseudo-random  number  generators  or  due  to  "noise" 

resulting  from  the  use  of  human  subjects  (as  in  INS),  the 
response  surface  of  the  simulation  will  itself  be  a random 
variable  which  will  have  associated  with  it  a variance  etc. 

One  reason  for  constructing  the  simulation  may  be  to  estimate 
the  variance  associated  with  the  response.  In  fact,  a similar 
sort  of  argument  may  be  found  in  Guetzkow  )1963;  117).  There 
is  no  general  a priori  reason  to  desire  low  between  run  vari- 
ance. • 

This  criticism  may  appear  trivial,  but  I think  that  it 
generalizes  too  much  of  the  INS  validation  effort.  That  is, 
reality  itself  (s)  is  seen  as  highly  internally  valid  in  the 
Hermann  sense  and  it  therefore  is  expected  to  have  very  low 
variance.  Since,  by  hypothesis,  reality  is  low  variance, 
the  external  validity  of  INS  ought  depend  in  large  part  upon 
its  ability  to  reproduce  reality.  This  position  should  become 
clearer  as  we  discuss  Hermann's  remaining  four  criteria. 

The  second  such  criteria,  face  validity,  "is  a surface 
or  initial  impression  of  a simulation  or  game's  realism  (1967: 
221)."  As  Hermann  points  out,  this  criterion  is  rather  vague 
and  is  generally  useful  only  in  early  stages  of  simulation 
development.  However,  recent  efforts  (Thorson  and  Phillips, 
1974;  Richardson,  1974)  suggest  that  face  validity  may  be 


It  might  be  counter  argued  that  in  the  case  of  stochastic 
computer  simulations  the  pseudo  random  number  seed  is  also 
an  input  value.  Therefore  we  would  want  to  have  low  between 
run  variance  with  the  seed  fixed.  However  this  argument 
loses  force  when  the  problem  of  internally  validating  the 
pseudo  random  number  generator  itself  is  considered  (see 

Mihram,  1972  pp  18-146). 
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capable  of  being  rendered  more  precise  by  identifying  the  mental 
images  of  individuals  who  "work  with"  the  process  being  repre- 
sented. 

The  notion  of  variable-parameter  validity  "involves  com- 
parisons of  the  simulations's  variables  and  parameters  with  their 
assumed  counterparts  in  the  observable  universe  (1967:  222)." 

In  terms  of  one  earlier  discussion,  we  are  concerned  that  the 
objects  of  m correspond  to  objects  in  s.  There  are  at  least 
two  problems  with  the  use  of  variable-parameter  validity  in 
evaluating  INS.  First  there  is  the  problem  of  aggregation. 

Even  a complex  simulation  such  as  INS  deals  with  variables  at  a 
highly  aggregated  level.  Thus  we  might  not  expect  the  INS  vari- 
ables to  have  simple  "real  world  counterparts."  In  fact,  unless 
we  are  willing  to  assert  a structure  preserving  morphism  between 
INS  and  "reality"  we  would  not  expect  such  a correspondence.  The 
earlier  quotes  from  Guetzkow  and  Hermann  suggest  that  no  struc- 
ture preserving  morphism  is  posited. 

If  we  are  not  willing  to  assert  a simple  correspondence 
between  INS  variables  and  those  of  the  real  world,  then  the 
problem  of  measurement  becomes  critical.  Measurement  theory 
deals  with  how  numbers  can  be  associated  with  attributes  or 
appearances  of  objects  in  such  a way  that  the  properties  of  the 
attribute  are  represented  as  numerical  properties  (see  Krantz 
et  al  1971).  The  problems  of  identifying  the  measurement  struct- 
ure necessary  to  discuss  variable-parameter  validity  in  highly 
aggregated  simulations  have,  to  my  knowledge,  not  been  discussed. 
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guess  is  that  simple  additive  structures  would  not  suffice. 

Yet  without  confronting  the  aggregation  and  measurement  pro- 
blems, it  seems  to  me  that  the  idea  of  variable-parameter 
validity  cannot  be  applied  to  INS  in  any  but  an  intuitive 
manner. 

The  fourth  standard  discussed  by  Hermann,  event  validity, 
uses  " 'natural*  events  as  criteria  against  which  to  compare 
outcomes  occurring  in  the  simulation  (1967:  222)."  For  exam- 
ple, Hermann  and  Hermann  (1967)  used  INS  in  an  attempt  to  sim- 
ulate the  outbreak  of  World  War  I.  The  primary  purpose  of  thei 
study  was  to  evaluate  the  validity  of  INS.  "One  means  of  inves 
tigating  this  question  is  to  ascertain  if  a simulation  produces 
events  similar  to  those  reported  in  a historical  situation 
(1967:  401)."  Thus  one  of  the  validity  criteria  being  used 
was  event  validity. 

However,  for  the  reasons  discussed  under  internal  validity 
such  a use  of  event  validity  requires  a commitment  to  a low 
variance  external  world.  What  if  the  "true"  probability  of 
the  specific  chain  of  events  leading  to  World  War  I is  .3? 

Would  we  then  want  to  say  that  a simulation  which  "reliably" 
reproduced  these  events  was  valid?  This  same  point  is  made  by 
Verba  (1964:  513)  when  he  notes  that  stochastic  "forces"  may  be 
operating  both  in  the  simulation  and  in  the  real  world.  Thus 
it  would  seem  that  high  event- val idity  is  neither  necessary 
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nor  sufficient  to  having  a "valid"  simulation. 

The  fifth  and  last  criterion  discussed  by  Hermann  is 
hypothesis  validity."  If  X is  observed  to  bear  a given  rela- 
tionship to  Y in  the  observable  universe,  then  X'  should  bear 
a corresponding  relationship  to  Y'  in  a valid  operating  model 
(1967:  223)."  This  criterion  would  seem  to  be  identical  to 
the  behavior  preserving  morphism  discussed  earlier.  Again, 
the  use  of  hypothesis  validity  requires  making  considerable 
measurement  assumptions.  These  assumptions  should  be  explicated. 

The  last  four  of  Hermann's  five  criteria  are  all  concerned 
with  assessing  the  descriptive  validity  of  simulations.  Each 
of  these  criteria  deals  with  certain  posited  correspondences 
between  the  refereent  system  and  the  simulation.  The  strongest 
of  these  correspondences  would  appear  to  be  the  behavior  pre- 
serving morphism  of  hypothesis  validity.  I have  suggested  that 
applications  of  these  criteria  to  INS  may  suffer  from  a deter- 
ministic (i.e.,  low  variance)  view  of  "reality"  and,  given  the 
aggregated  nature  of  INS,  a too  simple  view  of  measurement. 

These  criticisms  of  the  methodological  assumptions  under- 
suggest why  INS 

lying  the  evaluation  of  the  descriptive  adequacy  of  INS^has  not 
been  more  successful  as  a guide  to  programmatic  research  in  the 
development  of  scientific  theories  of  international  relations. 

By  this  I do  not  mean,  of  course,  that  INS  has  not  served  as 
a catalyst  for  a number  of  very  important  substantive  studies 
in  international  relations.  It  most  clearly  has. 
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However  one  of  the  luxuries  of  the  arm  chair  critic  is 
to  suggest  what  might  have  been  done  differently.  My  guess  is 
that  had  there  been  more  attention  paid  to  questions  of  aggre- 
gation and  the  related  problems  of  measurement  as  the  simulation 
was  being  constructed,  later  researchers  would  have  found  it 
much  easier  to  imbed  specific  research  questions  in  the  general 
INS  structure.  Some  work  we  have  done  at  Ohio  State  with 
Forrester's  World  Dynamics  simulation  strongly  indicates  that 
once  a complex  simulation  (especially  one  with  significant  non 
linearities)  is  constructed,  it  is  too  late  to  disaggregate  the 
concepts  and  relations  to  consider  specific  hypothses  using 
simple  measurement  structures.  The  tendency  then  is  to  do 
experiments  on  the  simulation  itself  and  to  compare  only  broad 
behavior  patterns  in  the  simulation  with  those  in  the  referrent 
system.  For  certain  purposes,  of  course,  this  is  perfectly  ade- 
quate. However  if  there  is  a need  for  a simulation  to  serve  as 
a "complete  processing  model"  with  which  to  investigate  the 
complexity  of  international  relations,  then  there  is  a need  to 
consider  early  problems  of  measurement  and  aggregation  and  dis- 
aggregation. Such  a purpose  would  seem  to  be  what  Guetzkow  had 
in  mind  when  he  wrote  that  simulations  "...permit  the  coherent 
amalgamation  of  subtheories  into  interactive,  hosistic  construc- 
tions of  great  complexity  (1969:  206)."  Further  he  (Guetzkow, 
1968:  210)  clearly  recognizes  the  measurement  problems  I have 
outlined. 
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Policy : A third  general  objective  according  to  which  INS 

might  be  evaluated  is  "policy."  There  are  numerous  ways  in 
which  INS  has  been  relevant  to  the  policy  community.  These 
include  pretesting  of  alternative  policies  on  INS,  providing 
a monitoring  system,  and  training  policy  makers  to  be  aware  of 
the  complexities  underlying  the  impacts  of  policies  by  involving 
them  in  INS.  As  many  of  the  papers  coming  out  of  the  INS  project 
have  pointed  out,  the  "validity"  of  INS  has  not  been  sufficiently 
established  to  rely  on  it  very  heavily  in  the  actual  policy 
making  process.  Indeed,  in  order  to  so  use  INS  we  would  prob- 
ably want  to  be  able  to  assert  at  least  a function  preserving 
morphism  between  the  simulation  and  the  referent  reality.  Yet, 
as  was  argued  earlier,  the  major  validation  efforts  have  cen- 
tered upon  establishing  only  a behavior  preserving  morphism 
(i.e.,  preserving  input-output  relationships). 

With  regard  to  the  policy  objective,  it  might  have  been 
useful  to  have  adopted  a more  explicit  control  theoretic  struc- 
ture. This  would  have  made  it  far  easier  to  address  questions 
of  system  optimazation  using  existing  techniques  (e.g..  Box, 

1954,  Box  and  Hunter,  1958,  Draper,  1962,  or  Gardiner  et  al , 
1959).  Had  this  been  done  it  would  be  (at  least  theoretically 
if  not  computationally)  possible  to  investigate  within  INS; 

(a)  the  relative  importance  of  alternative 
policies,  if  different  environmental 
conditions,  or  if  differing  parametric 
specifications  as  they  affect  the  similar 
response  at  some  point  T in  simular  time; 
and , 
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(b)  that  set  (or  combination)  of  policies, 

environmental  conditions,  and  parametric 
specifications  which  will  provide,  in 
some  sense,  the  optimal  similar  response 
at  time  T (Mihram,  1972;  402), 

Answering  (or  even  posing)  such  questions  would  probably 
require  reconceptualizing  INS  in  response  surface  terms  (this 
might  not  have  been  all  that  difficult  since  several  of  the 
"experimental  designs"  employed  in  INS  reports  approached 
doing  this).  More  importantly,  it  becomes  necessary  to  attach 
some  sort  of  objective  function  to  the  simulation  so  that  alter- 
native outcomes  can  be  ordered  with  respect  to  their  "desira- 
bility." The  problem  of  identifyir.g  such  functions  for  social 
systems  is  most  difficult  (for  an  analysis  of  the  problem  see 
Raiffa,  1969).  Yet  if  simulation  is  to  be  seriously  used  in 
policy  se] ection  it  would  seem  that  such  problems  must  be 
addressed . 
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Des ign : Whereas  with  the  policy  objective  the  concern 
was  with  identifying  and  implementing  feasible  strategies  to 
meet  some  goal(s),  design  problems  deal  with  identifying  and 
describing  various  mechanisms  for  the  achievement  of  goals. 

The  distinction  I am  making  here  between  policy  and  design  is 
analogous  to  the  distinction  between  the  values  of  variables 
(including  parameters)  and  their  structure.  Policy  changes 
are  changes  in  the  level  of  variables  and  design  changes  are 
changes  in  the  structure  relating  the  variables. 

Simulation  is  a very  powerful  technique  (i.e.,  means  of 
generating  data)  for  the  design  theorist.  The  low  cost  of 
computer  computation  makes  it  possible  to  examine  numbers  of 
complex  mechanisms  in  a variety  of  "environments."  Moreover, 
the  use  of  simulation  allows  the  investigator  to  deal  with 
non-linearities,  time  lagged  feedback  and  other  complications 
numerically  where  analytic  solutions  may  either  not  exist  or, 
if  they  exist,  be  beyond  the  symbol  manipulation  skills  of 
most  behavioral  scientists. 

Notice  too,  a key  distinction  between  the  policy  objective 
and  the  design  objective.  In  the  case  of  policy  the  represen- 
tation (i.e.,  mcM)  is  taken  as  being  fixed.  Our  concern  is  with 
the  impact  of  parametric  and  variable  value  (i.e.,  level)  changes. 
However,  in  the  case  of  design,  the  representation  itself  is 
the  datum  and  our  concern  is  with  identifying  desirable  (perhaps 
in  a constrained  sense)  elements  of  M.  Thus  experiments  designed 
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1 to  optimize  existing  simulations  will  not  generally  serve  the 

purpose  of  design  as  I have  outlined  it  here  (though,  of 
course,  it  may  give  us  an  idea  of  the  "best"  that  particular 
m can  do) . 

Moreover,  in  the  case  of  design,  simulation  can  be  quite 
properly  be  viewed  as  a data  generating  technique . The  data 
generated  are,  of  course,  the  performances  of  various  m M. 

Under  this  view  of  design,  questions  of  appropriate  methodology 
again  become  relevant.  Here,  for  example,  we  are  less  con- 
cerned with  "correspondences"  between  m and  s.  Rather  we  are 
interested  in  developing  preference  orderings  over  the  elements 
of  M.  As  Guetzkow  (1966)  suggests,  some  of  the  methodological 
( positions  of  "traditional"  utopian  thinkers  may  be  applicable. 

Ill  Conclusion 

The  title  notwithstanding,  this  essay  has  been  more  a 
critical  review  of  selected  aspects  of  the  methodology  of  INS 
than  it  has  an  "appraisal."  According  to  the  0. E. D. , to  appraise 
may  be  rewritten  "to  estimate  the  amount,  quality,  or  excellence 
of."  The  narrow  scope  of  this  essay  does  not  permit  an  ap- 
praisal. Moreover,  such  a task  requires  a more  experienced 
appraiser  than  I.  However,  and  this  may  be  more  revealing  any- 
way, I would  be  willing  to  bet  that  in  the  year  2000  it  will 
be  commonly  acknowledged  that  the  state  of  international  relations 
theory  owes  a great  debt  to  INS  related  activities.  Already, 
in  1974,  one  cannot  help  but  notice  the  number  of  rather  sophis- 
ticated projects  employing  simulation  (and  employing  it  as  a 


.1 


-29- 
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matter  of  course).  Compare  this  to  1964  or  1954.  Had  Harold 
Guetzkow  not  begun  the  Simulated  International  Processes  pro- 
ject, a large  number  of  us  would  be  unable  (for  a wide  variety 
of  reasons)  to  do  what  we  are  today.  Assuming  that  we  value 
that  which  we  are  doing,  how  can  we  do  other  than  consider 
INS  a success? 
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